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ABSTRACT 

The purpose of a research project was to identify 
approaches for improving the use of evaluation data through better 
linking of state-level evaluation and planning. The first phase 
included a review of literature in education and related disciplines 
and a national survey to elicit information from 55 states and 
territories concerning evaluation and planning linkages; 43 states 
responded. Although no single organizational structure category of 
the 12 organizational structures identified appeared best to 
accommodate planning and evaluation linkages, a model with separate 
boards for secondary and postsecondary education and with formal 
planning procedures or committees represented a deliberate effort to 
make planning and evaluation a more visible and coordinated process. 
The second phase involved determination of suggested strategies for 
improving planning and evaluation linkages. Findings were used to 
develop interview procedures for case studies in the third phase in a 
western, midwestern, and eastern state. Focuses were on ways in which 
planning and evaluation were mutually supportive and factors that 
encouraged and impeded linking of planning and evaluation. The 
research identified nine interrelated and mutually important factors 
as incentives for linking planning and evaluation. Planning' s impact 
on evaluation came primarily through formal assurances or guidelines. 
Evaluation had its impact on planning in program improvement, staff 
development, and complianct with federal regulations. (VLB) 
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INTRODUCTION 



Planning and evaluation arc typically viewed as two entities, each with its own 
language, practices, and professional communities. Vocational education is no exception to 
this phenomenon with planning and evaluation often earned out by different departments or 
different personnel within an agency (Asche, Strickland, & Elson, 1988). 

Federal vocational legislation has placed increasing emphasis on assessment, 
evaluation, and planning since the 1976 Vocational Education Amendments. States 
responded by developing a myriad of models, systems, and procedures for the generation. 
coUection, and analysis of labor market, enrolhncnt, programmatic, and foUow-up data. 
Most states have some approach to planning and evaluation. These approaches arc widely 
variable in scope and sophistication, and, usuaUy, the evaluation and planning functions are 
not symbiotic. Brannon (1985) summarized this evolution in requirements as a shift from a 
focus on quantity to an increasing focus on quality of vocational programs. Such 
requirements move beyond simple assessment (counting) and imply the need for a more 
interactive relationship between planning and evaluation functions. Since most states have 
some systematic approach to both planning and evaluation in place (Asche. 1985; Edington 
& Cruikshank. n.d.). one might assume "all is well" in tiie planning ami evaluation arena. 

Most planning systems are focused on operations and administrative planning, thus 
the evaluation systems tend to be compliance oriented (Asche. 1985). Consequently, 
functional links between comprehensive evaluation and substantive planning do not exist in 
most states (Strickland & Asche. 1987). Vocational education is not alone in its struggles 
witii linking relevant evaluation to the planning process as indicated by die rapidly growing 
literature base in the areas of evaluation utilization and innovation/change management. 
Such linking is particularly critical in vocational education, however, since tiiis field must 
constanUy meet the challenges posed by rapidly changing client populations, technology, 
funding, and state and federal policy initiatives. 

The purpose of this research, funded by the National Center for Research in 
Vocational Education (NCRVE). was to identify approaches for improving die evaluation 
utilization through better linking of state level evaluation and planning. The objectives were 
to assess the status of state-level planning/evaluation linkages, determine die incentives and 
disincentives to such linkages, and to identify linking strategies based on tiiis research. 
Further, the identified strategies were tested against the realities of state-level vocational 
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administration through three case studies. The final objective of this research was to 
develop and field review materials designed specifically to assist state vocational personnel 
in linking planning and evaluation within their respective states. An inscrvice program will 
be conducted during 1991 to introduce the training materials to state directors of vocational 
education and others involvcu in the planning and evaluation of educational programs. 



METHODOLOGY 
State Surveys 



The first phase of this research program included a review/synthenis of Utcraturc in 
education and related disciplines such as social changes, public adininistration, strategic 
planning, and evaluation (Asche. Strickland, & Elson. 1989). After completion of the 
review, a national survey was conducted to eUcit infMxnation from the states and temtones 
concermng evaluation and planning linkages. Particular care was taken to obtain data on 
activities for both secondary and postsecondar • vocational education and on interaction or 
articulation between these two levels. 

The review of literamre indicated that there is considerable variance in the ways 
vocational education is organized and administered within states. Two types of surveys 
were developed. Based on recent Uterature and interviews with a number of key federal 
and state vocational education officials, a preliminary screening survey was designed. The 
goal of this instrument was to obtain (1) information from state vocational directors 
regarding the organizational airangement for administration of secondary and 
postsecondary vocational education, and (2) the n^es. addresses, titles and telephone 
numbers of ti^e individuals responsible for planning and for evaluation at the secondary and 
postsecondary levels in each state. This instrument was mailed during the first quarter of 
1989 to fifty-five states and tenitories. With one loUow-up maiUng and telephone contacts, 
responses were received from fifty-one states/territories for a response rate of ninety-three 
percent The data derived from this instrument fomied the basis for all further contacts 
with the states. 

The primary data coUection instrument was designed to elicit basic information on 
how planning and evaluation are administratively organized, the procedures employed m 
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strategic and operational planning, and the mandated and optional evaluation activities 
conducted in each state or temtoiy. The draft instrument was reviewed by external experts 
and planning and evaluation personnel from six states. After modification, the areas 
included in the final instrument were the following: 

The location of vocational education within the state's CH-ganizational structure 
The identification of offices to which the persons responsible for planning and 
evaluation report 

3 . A description of how those responsible for planning cooidinate witii otiier agencies 
and departments. 

4 . A description of long-range planning procedures. 

5. A description of evaluation specifications included in the state plan. 
A description of the way evaluation affects the planning process, both fonnally and 
informally. 

A description of how sources, control, and allocation of ftinds affects planning and 
evaluation. 

The respondent's comments on how planning and/or evaluation could be improved. 

The first mailing, one follow-up mailing, and follow-up telephone calls yielded 
responses from forty-Uircc states for a response rate of seventy-eight percent Eleven of 
die responding states did not provide information for both the secondary and postsecondary 
levels. Since this was a qualitative type of insmiment. requiring data from multiple 
sources witiiin many of the states, tiiis response rate was deemed acceptable. Documentary 
analysis techniques were used to organize ihis information. 
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Case Studies 



The second phase of this project involved detemiination of suggested strategies for 
improving plamiing and evaluation linkages. Specific linkage components were identified 
and assessed along with the suggested strategies. These findings wert; used to develop 
inteiview procedures for die case studies in the third phase of Uiis research. Information 
obtained through die literature review and survey phase was used to prepare a portfolio of 
information on each site. Both the preliminary findings and the portfolio infomiation were 
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used to design a case study approach for gathering the necessary mformation to either 
verify the strategies or contribute to refinement or revision of the strategies. 

The states chosen for in^epth study by use of on-site interviews were selected on 
the basis of the review of literature and the state .esponses to both project questionnanes. 
Reconrntendadons from selected state directors of vocational education, resource persons m 
the U S Office of Adult and Vocational Education, and the National Council for 
Vocational Education were considered. An attempt was made to accommodate othe- . cto« 
such as complexity (population and federal funding level), geographic region of the 
country, and overall approach to administration of vocational education. As a result, three 
stctes were selected-a western state, a midwestem state, and an eastern state. 

Approximately three days were spem conducting the interviews in each state. 
Interviews were held with the following: the state director of vocational education, 
appropriate administrative personnel in the director's office, persons responsible for 
evaluation and planning for both secondary and pos.secondaiy levels, and other persons 
within the state's Departmem of Education and/or community college administration and 
governing boards. Interviews were also conducted with regional and local personnel 
responsible for planning and/or evaluation. Both individual and group interviews were 
held in each state. Tlie interviews were recorded for later transcription and analysis. 

Interviews were basically open-ended but structured around areas of concern 
derived from the project's earlier research activities. The central focus of the intet^iews 
was on ways in which planning and evaluation were mutually supportive, factors which 
encouraged linking of planning and evaluation, and factors which impeded such linking at 
the secondary and postsccondary levels. 

Data available from the state survey and the case studies was analyzed and 
examined against the proposed planning/evaluation linkage strategies. Results from this 
latter phase of analysis and refinement of the Unkage st«tegies wUl constimte *e substance 
for development of a guide for use by vocational education planning and evaluation 
personnel. TTie guide will be reviewed by planning and evaluation specialists and will be 
revised as needed. 



9 



ERIC 



CURRENT STATUS OF PLANNING AND EVALUATION LINKAGES IN 
STATE DEPARTMENTS OF EDUCATION 



Results from a Survey of States 

Organizational Structure 

One of the more basic questions of the survey had to do with the actual location of 
secondary and/or postsecondary vocational administration. While over a dozen 
organizational structures were identified, five major structures seemed to account for the 
majority of state administrations. The first structure is characterized by two separate boards 
(see Figure 1-a) — one primarily for secondary education and one primarily for 
postsecondary education. In most instances, the secondary board acts as the sole state 
agency for receiving fiinds from the Carl D. Perkins Vocational Education Act of 1984. In 
some states, the federal funds are received by the postsecondary board. Twelve states 
(twenty-eight percent) responding to the survey evidenced this form of wganization. 

The second category evidenced two variations of one basic structure. In this 
structure category, vocational administration ultimately answers only to a state board of 
education (see Figure 1-b). A separate board or system for iK)stsecondary education is 
present in the state, but no vocational programs come under the jurisdiction of this boanL 
Under the board of education, the administration of vocational education is housed within 
the department of education. Responsibility for secondary and postsecondary vocational 
programs may be combined or separated into two units. This organizational strucmre was 
identified for eleven (twenty-six percent) of the survey respondents. 

In Figure 1-c, administration of vocational education (both secondary and 
postsecondary) is housed in a unit or agency separate from the department of secondary 
education and other agencies answering directiy to the state board of education. As in the 
second category, a separate board or system for postsecondary education is present in the 
state, but no vocational programs come under the jurisdiction of this board. Five states 
(twelve percent) reported administrative structures characterized, by this category. 

The fourth major category appears very similar to the first category. The major 
difference is the presence of a formal or legislated committee set up to link the planning 
functions for se':ondary and postsecondary programs (see Figure l-d). This strucmre is 
becoming an increasingly popular form of vocational administration as several respondents 
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indicated that progress is being made within their states to move to this structure. While 
Perkins monies still flow primarily through the secondary agency, substantive input and 
exchange characterize the allocation and planning of programs at both levels. In some 
cases, all providers of vocational education (including Job Training Partnership Act [JTPA] 
and community-based organizations) are involved in the joint planning/evaluation process. 
Seven states (sixteen percent) were grouped in this category. 

The fifth category included four states (nine percent) that reported administrative 
structures characterized by a separate state board for vocational education (see Figure 1-e). 
T^as board governs all vocational programs, both secondary and postsecondary, separate 
from other boards responsible for either secondary or postsecondary education. The state 
dir^tor for vocational education reports directly to the State Board of Vocational Education 
(SBVE). 

While these major categories collectively comprise most state structures, some 
states evidenced structure, iliat were more or less individually unique. The remaining four 
state respondents, consequentiy, were not included in the major structure categories 
described above. 

Some structures appear to lend themselves more to fiscal as opposed to 
programmatic interagency relations. For instance, the structure illustrated in Figure 1-a is 
often characterized by a fiscal approach to planning/evaluation with the flow of funds from 
the sole state agency to the other board constituting the primary formal contact between 
secondary and postsecondary vocational education. The remaining structures provide at 
least a structural linkage between secondary and postsecondary vocational education and 
suggest more programmatic as opposed to fiscal exchanges. On the other hand, in those 
structures where vocational administration is housed within larger departments, the 
integration of vocational planning and evaluation with that for secondary/postsecondary 
education as a whole becomes an issue. In this latter context, a tendency toward 
compliance rather than strategic planning is observed. 

While no one structure category appears to best accommodate planning and 
evaluation linkages, the model illustrated in Figure 1-d represents a deliberate effon to 
make planning and evaluation a more visible and coordinated process. Also, the tendency 
for some states to restructure in order to (1) effect coordination between secondary and 
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postsccondary sectors, (2) enable rcgionalization of planning and evaluation procedures, 
and (3) allow for a more direct flow of planning/evaluation information into the policy or 
decision making context, suggests that administrative stnictunss may be a tool by which to 
facilitate move effective planning and evaluation practices. 

Responsibility for Planning and Evaluation 

The organizational structure for vocational education administration is the primary 
factor governing die responsibility for planning and evaluation. Specific information 
involving the network of responsibility for planning and/or evaluation was limited. 
Planning seems to be independent of evaluation in terms of administrative location and 
personnel responsible for each function. 

Coordination with Other Agencies 

In ail states reporting, the minimum level of coordination between agencies met the 
requirements of the Perkins Act. Some of these planning efforts include interagency 
committees, the State Council for Vocational Education, and a state planning committee. 

Long-Range Planning 

Long-range planning is perceived by the majority of respondents as that process 
required by the Perkins Act. Beyond this process, a number of states have attempted to 
implement a variety of approaches to long-range planning. One common element among 
these states was the instimtion of a task force or interagency committee to review the goals 
of vocational education priOT to initiating a long-range or strategic planning pnx:css. 

Specifications for Program Evaluation 

Most state plans or long range plans include specifications for program evaluation. 
The nfiinimum requirement for evaluation in most state plans was that it would be completed 
in compliance with federal legislation. Two common elements identified were a five year 
time span for evaluations (due to the requirements of the Perkins Act) and a team approach 
to on-site visitations. Thirteen respondents indicated a move toward use of outcome criteria 
or indicators. 




Impact of Evaluation on Planning 

Responses relevant to the perceived impact of evaluation on planning generally fell 
into five categories. Of the forty-three states responding, some indicated that evahj-^tion 
impacted on planning in more than one category. The five categories arc sununarized as 
follows: 

1 . No impact. At this level there is limited, if any, utilization of evaluation results in 
the planning process (four states), 

2. Local-level planning. Local evaluation is utilized in local planning processes. In 
some instances, the local evaluation is sent to the state level for inclusion in the state 
plan. Local planning can be affected by a state evaluation in lieu of a local 
evaluation. Evaluation is critical to program improvement efforts, most often 
providing the basis for development of local action plans or funding of 
improvement initiatives (twenty-seven states), 

3 . Funding support. Evaluation serves to support funding or refunding of programs, 
courses, or vocational projects at the local, regional, and state levels (six states). 

4. Staff development. Evaluation rcpons are used to raise the awareness of 
administrators with regard to various vocational issues. Another aspect of staff 
development is in the form of sharing strategies for improvement of vocational 
programs. Experiences across localities are shared or communicated to allow 
broader application of programs or initiatives (two states). 

5. Compliance, Evalupions serve in support compliance with federal and state 
regulations. Conducting local evaluations satisfies requirements by the state and 
assures localities of continued operation or funding. Similarly, state level 
evaluations are required to satisfy Peridns Act regulations (six states). 

Source, Control, and Allocation of Funds 

Response levels to this particular item in the survey were quite low. Those states 
that did respond provided widely divergent examples of the effect of funding on planning 
and evaluation such that these responses had to be considered unique to each state. 
Consequently, the role of source and allocation of funds in shaping evaluation and planning 
processes must be pursued in future research efforts. 
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Improving Planning and Evaluation 

Seven common themes for improvement of planning and evaluation were identified 
by respondents: 

1 . Provide more staff, time, and money to properly conduct planning and evaluation. 

2 . Provide guidelines for indicators/measurements of inputs and outcomes. 

3 . Require better preparation and use of the evaluation results. 

4 . Inclement a regional approach to administration. 

5 . Improve the needs assessment procedures. 

6 . Improve data collection procedures and the quality of the collected data. 

7. Develop more coordination and cooperation among state agencies and between all 
levels of administration. 

Respondents from nine states noted that improvements in the administrative 
process, including planning and evaluation, are underdevelopment at this time. 



Synthesis of Results 

Several portions of the survey inquired about the interaction of planning and 
evaluation activities and/or personnel. Consequently, these items were examined 
collectively in order to obtain a comprehensive perspective of planning/evaluation linkages. 
Two general models of planning/evaluation practices were discerned in this phase of 
analysis. One served to suppon the more programmatic needs of planning/evaluation; 
while the other served to support the fiscal needs of planning and evaluation. 

Thc programmatic model involved data collection and/or self-evaluations by 
teachers and/or administiutors at the local levels. Enrollment or follow-up data most often 
were aggregated in some form at the state level. This data, along with the self-evaluation 
data, was used in the on-site visitation phase. The visitation teams could be comprised of 
teachers or administrators within the locality, or, more likely, of individuals from the state 
department of education or individuals designated as "peers" from otiier localities. A 
number of states accomplished the on-site visit through the accreditation teams set up to 
determine local schools' eligibility for either state or regional accreditation. Findings or 
recommendations from these teams were combined with data from the information phase to 
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provide the formal working report. Based on this report, a plan for improvement or an 
action plan was developed. Monitoring of this plan was conducted by state staff who 
typically also provided technical assistance as needed. In most states, funding as necessary 

and as available — was provided to support the recommendations outlined in the 

evaiuadon report 

While the fiscal model had implications for programs, the emphasis was clearly 
within the allocation process. In this process, localities make application for various 
courses, programs, and projects. As part of the application process, an evaluation plan is 
required. Generally, this plan is developed within guidelines set by the state. It is the 
responsibility of the applicant to carry out the evaluation as described in their application for 
funds. A state level panel or agency reviews the application and evaluation plan. Funding 
for the application is determined and may be rejected if an appropriate evaluation plan is not 
provided. Refunding of applications depends on supporting evidence from evaluations 
done in the initial applications. 

While there are some modifications of die above models of planning/evaluation 
interaction, very littie middle ground exists between tiiem. It is clear tiiat two perspectives 
on vocational program planning and evaluation exist: a view directed toward program 
improvement and a view directed toward program accountability. The interaction of 
planning and evaluation processes appears to serve as die defining medium for which 
perspective prevails at a given point in time. 



SUCCESSFUL STATE PLANNING AND EVALUATION EFFORTS 

Three Case Studies 

Western State 

"A formal, yet ftiendly, partnership between public schools and colleges" described 
the regional planning and evaluation process in a western state according to a local 
vocational director. The process included all educational institutions from die local school 
districts to die research university. A state staff member indicated Uiat the planning process 
was affecting policy direction by setting the tone for policy development. 
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Over a three-day period, the project staff conducted interviews with individuals 
from the State Department of Education, State Board of Regents, regional personnel, and 
local school district personnel. All of those interviewed were enthusiastic about the 
planning process. 

The Department of Education, mandated by State Code, initiated a formal planning 
cycle in 1984. Following a two year process, the Department developed the 1988-93 
Vocational-Technical Education Master Plan as the second five-year plan in the cycle. 

Rcgionalization wis the key word in the plan. There were nine regions. Each 
region included a community college and/or a four-year college/university and all 
surrounding public schools. Some regions included an area vocational center. Local 
education agencies did not submit plans to the state department for approval. Instead, each 
local education agency and the college/university presented their plans to a thirteen member 
Regional Vocational-Technical Planning and Coordinating Committee. The Committee 
included a representative ftiom each type and level of occupational program provider in the 
region. The Committee developed annually the regional vocational-technical plan. 
Regional labor market data guided the planning process. Representing the Regional 
Coordinating Committee, the chief executive officers of the school districts, the area 
vocational center, and the college or university presented the Regional Plan to the State 
Board of Education/State Board of Regents Liaison Committee for review and comment. 
After review, the chief school executive officers of the region presented the plan to the State 
Board for Vocational Education for approval. 

The regional concept of planning reduced unwarranted duplication within the 
region. Each local provider of an occupational program must justify the program based on 
labor market data. Labor market data may indicate the need for a new program in the 
region. The regional plan must include a proposal for the new program. If the completers 
of an occupational program are not being placed, based on the results of a follow-up 
survey, the plan must include steps to phase out the program. Nonoccupational programs, 
such as exploratory or consumer and homemaking programs, arc not judged on the basis of 
placement data. 

While planning procedures were well documented, fonnal evaluation procedures 
were not spelled out in the Master Plan, except for some mention of Perkins Act 
requirements. This is not to imply an absence of evaluation in the planning process. 
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Secondary schools usually participated in the Northwest Region Accreditation program. 
The State Department of Education was mandated to serve as the sole state agency for 
vocational education to accredit area vocational centers and colleges. A third-party 
evaluation conducted by an out-of-state consultant evaluated twenty percent of the 
vocational programs each year. The consultant used existing data and visited selected 
vocational programs. State personnel indicated that it was very important to have an 
"outside" person evaluate the programs. Informal evaluation procedures were very much 
in evidence at all levels. Continual evaluations took place as a part of local advisory 
committee deliberations and consultation services provided by state program specialists. 

The major limitations to full implementation of the Master Plan, expressed by 
respondents, centered around two factors. The first was "turf." In some regions there was 
less than fuU co<^)cration/cooniination between vocational services and between educational 
agencies. The second factor, closely related to tiie first, was the lack of leadership in those 
regions. With time and assistance from the State Department of Education, full 
implementation of the Master Plan will be possible. 

An associate superintendent from the Department of Education indicated that the 
planning and evaluation process resulted in improved vocational education. First, it was 
having a dynamic impact by emphasizing the worth and value of individuals who were not 
college bound Second, tiicrc was a positive, cooperative working relationship between all 
educational agencies. Third, the Master Plan resulted in a state and regional focus on 
vocational education. 

Midwestern State 

Those interviewed in tiie midwestem state viewed planning and evaluation as two 
factors within the same phenomenon. The project team interviewed university faculty. 
State Department of Education personnel, State Board of Community College personnel, 
regional system directors, and local vocational directors. 

The midwestem state has a much higher population density and thus, more public 
schools and community colleges than the western state just described. Regionalization was 
an important part of this state's planning and evaluation process. 

Education for Employment (EFE) began with a comprehensive policy study 
mandated by the State Board of Education in 1982. The overall goal of EFE was the 
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revitalization of vocational education. One of the conclusions from the study laid the 
foundation for a regional system of high schools and area vocational centers, and, in turn, 
created an '^alliance" with their respective community colleges. The boundaries for the 
regions were not drawn in common with the thirty-nine community college regions. The 
result was the establishment, authorized by State Code, of sixty-one regional systems by 
the State Board of Education. According to comments from both secondare and 
postsecondary interviewees, this lack of common boundaries hindered the develo; v 't and 
implementation of vocational education policy. 

In most regional systems, the board of control included the superintendents of the 
local high school districts. The local community college dean was a voting member on 
only four regional system boards. Each region had a full-dme director. A local school 
district wanting either state or federal funds must participate in a regional system. 
Coordination with other vocational education providers such as JTPA was required, but 
representatives from such groups did not serve on the board. 

The regional systems resulted in a substantial strengthening of the link between 
secondary schools and between secondary and postsecondary vocational education. Some 
regions are developing long range plans based on the current state quality indicators. 
Strategies were also developed by the regional groups to cany out the plans. 

Focus group sessions were one strategy used by the State Department of Education 
to keep in touch with what was happening at tiie local level. These sessions were held 
throughout the state to receive input concerning which direction vocational education 
should be taking in the state. 

The state had an extensive evaluation procedure in place for many years. With the 
advent of EFE, a new procedure was being developed which relied on the microcomputer 
for a major part of the data analysis activities. More emphasis was placed on quality 
indicators rather than on process oriented factors. The quality indicators, or "vital signs," 
included labor market justification, enrollment, placement and continuing education, 
employer satisfaction, student satisfaction, general employability skills, and cost. 

One interviewee expressed some reservation with this system. This individual 
believed that the system was too traditional in its approach to evaluation for the 1990s and 
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that the emphasis should be on the perfonnancc of the students — what they know, what 
they can do — and not on the satisfaction of the students and employers. 

As in the western state, labor market information served as one of the quality 
indicators. One individual noted his reluctance to rely on labor market information to judge 
a program. Labor market information may be missing what is happening in industry. He 
gave the example of welding programs and facilities being closed because of negative labor 
market information, yet, several different occupations need welding as part of the total 
training package. Follow-up of former students was a vital component of evaluation. 
Determination of the labor market advantage of a former vocational student was critical. 

This state had a mandated planning process, as did the western state. Evaluation in 
the formal sense was not used extensively at the time of the interview. Once the new 
evaluation system is on line, formal evaluations are expected to be used at both the 
secondary and postsecondary levels. Informal evaluations were evident at all levels based 
on the conunents given in the interviews. 

Eastern State 

The eastern state is a large, heavily populated state with diverse populations and 
large urban areas. The State Department of Education has historically exerted strong control 
over education, prescribing curricula for elementary and secondary schools. The 
governance of vocational education was not distinct from academic programs at the highest 
levels in the state hierarchy. Secondary vocational and academic programs were closely 
linked together since they both fall under the direction of the Office of Elementary and 
Secondary Education. The 1988 reorganization of the State Department of Education 
resulted in the creation of an Office of Continuing Education, within the Office of Higher 
and Continuing Education, to serve adult populations. Thus, postsecondary vocational 
education was subject to the same standards as other postsecondary programs. 

Many of the vocational education programs were in the two-year colleges. 
Although there were linkages between vocational and academic programs within secondary 
and postsecondary education, formal articulation did not exist between secondary and 
postsecondary agencies. 

The mission of volitional education was clearly stated in 1984 after the completion 
of an extensive and inter -.ve two-year review of orcuparional and practical arts programs. 



Nearly two hundred representatives from business and industry, social sciences, students, 
teachers, and administrators met to determine the purposes and programs for occupational 
and practical arts education through the remainder of the century. Those findings set a clear 
mission and new direction for vocational education. The key areas identified by this review 
have driven subsequent local and regional evaluation and planning activities. 

Evaluation and assessment of needs formed the basis for planning. Local plans 
were developed from several sources including evaluation reports, recommendations of the 
local advisory council, sex equity action plans, labor market demand* student interest, and 
demographic changes. The state handbook described the plans as blueprints for linking 
state and federal resources. 

The program improvement process built upon the evaluation process. It stemmed 
from the idea that, once the evaluation identified local needs, agencies would be able to 
develop plans to improve vocational programs. The program improvement process 
evolved into skills for program improvement. It was discovered that what people needed 
most were skills to enable them to work witii other people in their agencies to address the 
evaluation needs. 

The smicture of the state vocational organization focused on function with service 
areas combined to achieve a more comprehensive perspective. A state-level individual 
described the structure as follows; 

We shifted from individual plans to cooperative planning last year where we 
asked agencies to put together service area planning groups that would look 
at a particular geographic area and then assess the needs of the area and have 
a number of members participating in establishing priorities for services a 
student would get 

Connections between evaluation and planning existed both formally and informally. 
Plans submitted for state approval must include evaluation reports which facilitate the 
linking of planning and evaluation. State personnel believed the inclusion of evaluation 
data had positive effects on vocational education. In particular, it helped vocational 
educators obtain legislation for vocational education act dollars. The Perkins Act limited 
the availability of funds for equipment and, tiiercfore, agencies became more dependent on 
state money. Since the evaluation data was included in the annual plans, the supervisors 
and others who reviewed annual plans were able to see exacUy what was needed and what 
was planned. One interviewee noted the importance of linking planning and evaluation: 
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We've tried to link them [planning and evaluation] very closely over the 
years. We've included the evaluation report in the plan. There's a needs 
assessment section in the plan that requires them to use the evaluation data 
as well as other information. 

Access to evaluation material was seen as a positive benefit to vocational programs. 
A specific example was given by one state department official: 

We were able to use the data fmm the evaluation to present to the board and 
then [we had] even better evidence that wc needed to move... .We were able 
to restpiwoire our whole guidance and placement service there; hire social 
workers, civil service placement people, and new counselors; and just 
redire:t the whole thing. 

In addition to formal written requirements that made evaluations part of local plans, 
the assignment of responsibilities among state personnel linked evaluation and planning in a 
different way. Overall, planning and evaluation were conducted by a professional staff 
who were well trained in the areas of planning, evaluation, and research. Often individuals 
performed more than one of these functions. Although the impact of holding responsibility 
for both functions was not easily measured, awareness of the functions and procedures 
involved in both processes strengthened the commitment to linking them. 



IMPLICATIONS FOR PLANNING AND EVALUATION LINKAGES 

The original plan for this project called for the development of one or more models 
for linking planning and evaluation in vocational education. It became evident as the 
research activities progressed that there is considerable variance in the ways vocational 
education is organized and administered within the states. The organizational structure is a 
primary factor in governing the agency and the personnel responsible for planning and 
evaluation. In many cases, planning is completely independent of evaluation in terms of 
the administration and the personnel responsible. Other factors which are widely divergent 
from state to state are die source, control, and allocation of funds to conduct planning and 
evaluation activities. It was evident in reviewing the existing literature and the analyses of 
the survey responses and interview transcripts that there is no single pattern that fosters 



18 



22 



linkages between planning and evaluation. Therefor .e idea of developing a theoretical 
model(s) was abandoned. 



Factors Important in the Linking of Planning and Evaiuation 

The results of this work suggest that linkages occur in different ways within 
different state structures. However, through the survey and case study visits, nine factors 
were discerned as incentives for linking planning and evaluation. These factors are 
illustrated in Figure 2. In particular, two characteristics are demonstrated through this 
illustration: the intenelat^ness of the factors and the mutual importance of each factor. 
For instance, leadership is frequently necessary for development of a clear policy or 
mission, yet, in some cases, the development of a clear statement of vocational education's 
mission was the catalyst needed to allow for strong leadership. The two factors are 
interrelated, and, depending on the context, both are important as initiators of effective 
linkage. The remainder of this section further elaborates on the interaction of these factors 
as observed in the surveys and case studies. 

The factors of in-state regionalization and interagency cooperation lend themselves 
very well to facilitating linkages between planning and evaluation. It is suggested that 
regional boundaries be based on community college service areas. Regionalization includes 
public schools, community colleges, and other providers of vocational education and 
should result in articulation between all levels of program deliverers. Tlie involvement of 
all local providers allows them to buy into the process and thus have a commitment to it. 

Implementing regionalization and cooperation may cause concern for local officials. 
As one local administrator noted there were 

feelings of turf protection or there were some underlying motives at the 
college; ...they want something from us, or they're in trouble right now and 
they can use us, but, as soon as they are out of trouble, they're going to 
drop us. 

This rfficial noted, however, that regionalization "was a godsend because it brought 
together independent school districts and a college into a very formal and friendly 
pannership in wanting to do things together." 
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In all of the three states selected for on-site interviews, planning was done on a 
regional basis, yet mandates differed from state to state. As each case study illustrated, 
there was no single pattern that fostered linkages between planning and evaluation. 
However, to achieve regionalization and interagency cooperation requires a clear policy 
direction or mission and a commitment to that mission by all involved, especially state 
departments of education. In two of the states, a mandate from the state board of education 
initiated the process. Moreover, leadership appears to be a major asp«.ct in initiating action 
on the above factors. It is critical that the leaders be professionally trained in the processes 
of planning and evaluation. 

When present, coordination of programs between secondary aikd postsecondary 
vocational education was a tremendous asset to linkage of planning and evaluation, and, 
when absent, it was a definite limitation. In son^ cases, this coordination began with 
development of a data system and joint planning activities. On the other hand, changes in 
educational policies spurred increase articulation between secondary and postsecondary 
delivery systems. Increased activities in this area seemed to have renewed an interest in 
development of usable and accessible data systems and joint planning/evaluation activities. 
Planning must look to evaluation for information and, in this sense, directs the evaluation 
process. 

Finally, one element contributing to effective linkage — an emphasis on 
comprehensive vocational education— came more from on-site observation than from the 
survey results. Vocational education has several program areas of instruction which can 
exhibit varying levels of strength, activity, and cooperation. Attempts to regard vocational 
education from a more comprehensive perspective have the advantage of being able to 
detect means for renewed programs and delivery systems for all of vocational education. 

There are several disincentives that will hinder attempts at linking planning and 
evaluation witiiin an educational system. All of the incentives discussed earlier need not be 
present, but their absence tends to inhibit the linking process. In addition, the following 
factors serve as disincentives to the linkage process: 

• Tradition and maintaining the status quo. 

• Difficulty in taking risks in administrative planning. 

• Problems in understanding the complete benefits of linking planning and 
evaluation. 
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• Demographic differences across a state's geography. 

• Time needed to build trust between agencies, businesses, and individuals. 

• Lack of similar evaluation procedures, philosophies, and context of secondary and 
postsecondary educational procedures. 



Impact of Planning and Evaluation 

Overall, the evidence of planning's impact on evaluation suggests a decidedly 
dctachrd relationship. Impact comes primarily through formal assurances or guidelines. 
Local evaluations are required through fundingA^funding requirements, and state 
evaluations are required to satisfy Perkins regulations. These requirements are Included in 
state plans, but the need or purpose these evaluations should serve vis a vis planning 
program development or improvement is virtually nonexistent A direct impact of planning 
on evaluation has to do with the more recent requirement through state/local plans to 
analyze supply/demand and enrollment data. While these activities may be construed 
simply as accountability tactics, there is some focus on goals being set for evaluating 
programs — goals which have implications far future planning and evaluation activities. 

Evaluation had its impact on planning typically through an interactive state/local 
system. This involved either local evaluations fed into the development of a state plan, or a 
state evaluation fed into development of local plans. Evaluation was critical in program 
improvement efforts, most often providing the basis for development of local action plans 
or funding of improvement initiatives. By the same token, evaluation served to support 
funding or refunding of programs, courses, or vocational projects. 

Another area of impact for evaluation focused on staff development. Evaluation 
reports were used to raise the awareness of administrators with regard to various vocational 
issues (e.g., delivery of programs, articulation of secondary/postsccondary programs, and 
placement of program completers). Another aspect of staff development came in the form 
of sharing strategies for improvement of vocational programs. Experiences across 
localities were shared or communicated to allow broader application of programs or 
initiatives as applicable. 
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Evaluations also served to support compliance with federal regulations. 
Conducting local evaluations satisHed requirements by the state and assured localities of 
continued operation or funding. Also, state level evaluation repons were required to satisfy 
Perkins regulations and assure compliance with federal legislation. 
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